Apparatuses and methods for color matching and recommendations

ABSTRACT

An image or a spectrum of a surface may be acquired by a computing device, which may be included in a mobile device in some examples. The computing device may extract a measured spectrum from the image and generate a corrected spectrum of the surface. In some examples, the corrected spectrum may be generated to compensate for ambient light influence. The corrected spectrum may be analyzed to provide a result, such as a diagnosis or a product recommendation. In some examples, the result is based, at least in part, on a comparison of the corrected spectrum to reference spectra. In some examples, the result is based, at least in part, on an inference of a machine learning model.

BACKGROUND

Analysis of surface color is used in a variety of industries. Forexample, surface color may be analyzed for recommendations of cosmeticproducts or paint colors. For example, any consumers choose skinfoundation based on a visual comparison. However, this requires the userto apply several different cosmetic products which may be messy and timeconsuming. In another example, to identify the correct paint to use fora wall a consumer may need to acquire paint swatches from a store,manually compare the swatches to the wall, then return to the store topurchase the proper paint or acquire additional swatches if none of theprevious swatches provided an acceptable match. Acquiring data toquantitatively analyze a color of a surface and provide a productrecommendation based on the analysis may require specialized equipmentand a well-controlled environment to reduce lighting interference. Thismay reduce the availability and/or increase the cost of analyzingsurface color. Accordingly, techniques for analyzing surface color withless expensive and/or more readily available equipment may be desired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of a system arranged in accordanceexamples of the present disclosure.

FIG. 2 is a schematic illustration of a computing system arranged inaccordance with examples of the present disclosure.

FIG. 3 is an example of a mobile device in accordance with examples ofthe present disclosure.

FIG. 4 illustrates example spectral plots in accordance with examples ofthe present disclosure.

FIG. 5 illustrates a method in accordance with one embodiment withexamples of the present disclosure.

FIG. 6 is a flowchart of a method in accordance with examples of thepresent disclosure.

FIG. 7 is a flowchart of a method in accordance with examples of thepresent disclosure.

DETAILED DESCRIPTION

The following description of certain embodiments is merely exemplary innature and is in no way intended to limit the scope of the disclosure orits applications or uses. In the following detailed description ofembodiments of the present apparatuses, systems and methods, referenceis made to the accompanying drawings which form a part hereof, and whichare shown by way of illustration specific embodiments in which thedescribed apparatuses, systems and methods may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice presently disclosed apparatus, systems andmethods, and it is to be understood that other embodiments may beutilized and that structural and logical changes may be made withoutdeparting from the spirit and scope of the disclosure. Moreover, for thepurpose of clarity, detailed descriptions of certain features will notbe discussed when they would be apparent to those with skill in the artso as not to obscure the description of embodiments of the disclosure.The following detailed description is therefore not to be taken in alimiting sense, and the scope of the disclosure is defined only by theappended claims.

Surface color analysis is an emerging area of interest to provideimproved color matching such as in cosmetics and painting. For example,in cosmetics, the lighting environment and/or the shininess of skin maylead to misjudgment of the skin color, resulting in a poor productselection. A store associate may recommend a foundation product based onthe appearance of a customer's skin tone at the store, However, when thecustomer leaves the store, the skin tone may appear different because ofthe change in lighting. The associate may have recommended a differentfoundation product if the associate and customer had met outside thestore under different lighting conditions. Existing techniques foranalyzing surface color may require specialized equipment, which maylimit the ability to acquire surface color data and/or providerecommendations based on analysis of the surface color. Therefore,techniques for a more affordable and/or reliable surface color analysisis desired.

According to examples of the present disclosure, a mobile device may beused to acquire surface color data and extract a measured spectrum fromthe data. In some examples, the mobile device may further analyze themeasured spectrum to compensate for ambient light influence on themeasured spectrum and generate a corrected spectrum. In some examples,the mobile device may provide a product recommendation and/or adiagnosis based at least in part on the corrected spectrum, In the casewhere the surface is a user's skin, a product recommendation may includea cosmetic product (e.g., base, foundation, setting powder, concealer)and a diagnosis may include a condition (e.g., redness, pores,pigmentation, oily or dry skin). In another example, the surface may bea wall and a product recommendation may include a paint color thatmatches the color or is complimentary to the tone of the surface color(e.g., warm, cool, neutral). In another example, the surface may be anitem of apparel and a product recommendation may include matchingclothing articles and/or accessories (e.g., shoes, sunglasses,necklaces, etc). In some examples, the product recommendation and/ordiagnosis may be generated by a machine learning model based, at leastin part, on the measured spectrum and/or user's feedback on the productrecommendation and diagnosis. As used herein, machine learning referscollectively to machine learning, deep learning, and/or other artificialintelligence techniques for making inferences from data.

FIG. 1 is an illustration of a system 100 arranged in accordanceexamples of the present disclosure. The system 100 may include aprocessor 102, a computing device 114, a spectrometer and/or camera 104,and a light source 106. In some examples, the system 100 may alsoinclude a cloud computing device 110. The system 100 may acquire animage and/or a spectrum of a surface 112 using a camera and/orspectrometer 104. In some examples, the surface 112 may be illuminatedby ambient light (Ia). in some examples, the surface 112 may bealternatively or additionally illuminated by light (Is) from the lightsource 106 and the spectrometer and/or camera 104 may acquire the dataassociated with the surface 112 (I). The data may include an imageand/or a spectrum of the surface 112. The data may be processed by theprocessor 102 to extract a measured spectrum (I). For example, theprocessor 102 may extract a measured spectrum from an image acquired bythe camera 104 or the measured spectrum may be acquired directly fromthe spectrometer 104, However, in some applications, the measuredspectrum may not accurately reflect the color of the surface 112, forexample, due to effects of ambient light 108. Ambient light 108 may havean uneven and/or unknown spectrum that may affect the apparent color ofsurface 112. Ambient light 108 may be reflected off the surface 112and/or other surfaces and collected by the camera and/or spectrometer104. Thus the measured spectrum (I) extracted by the processor 102 maybe a mixture of the spectrum of the surface 112 and the ambient light108. Therefore, the processor 102 may analyze the measured spectrum toremove ambient light influence (Ia) from the measured spectrum togenerate a corrected spectrum of the surface 112 (I0). That is, in someexamples, the corrected spectrum that includes spectrum of the surface112 may be expressed as I0=I−Ia. In examples where light source 106 isused to illuminate the surface 112, the corrected spectrum may beexpressed as I0=I−Ia−Is.

The computing device 114 may compare the corrected spectrum (I0) with adatabase of reference spectra to generate a result. The result may be aspectrum from the data base of reference spectra that is the closestmatch to the corrected spectrum of the surface 112. The computing device114 may provide a product recommendation and/or a diagnosis of thesurface based, at least in part, on the result.

In some examples, the computing device 114 may be optionallycommunicatively coupled to the cloud computing device 110. For example,processor 102 may be communicatively coupled to the cloud computingdevice 110. In some examples, the cloud computing device 110 may storedata including images and/or spectra received from computing device 114,a measured spectrum, a corrected spectrum generated by the processor102, one or more databases of reference spectra, the result generated bythe processor 102, and/or user feedback. In some examples, the cloudcomputing device 110 may implement one or more machine learning modelsto make inferences based on the data provided by the computing device114. For example, the processor 102 may provide the corrected spectrumto the cloud computing device 110 and the cloud computing device 110 mayreturn a product recommendation to the computing device 114 based on aninference of the one or more machine learning models.

FIG. 2 is a schematic illustration of a computing system 200 arranged inaccordance with examples of the present disclosure. The computing system200 may include a mobile device 222, a computing device 218, and/or alight source 208. Optionally, in some examples, the computing system 200may include a display 204, camera 212, a spectrometer 210, and/or acloud computing device 230. In some examples, the computing device 218may include one or more processors 206, a computer readable medium (ormedia) 224, a memory controller 220, a memory 216, and/or userinterface(s) 214. The computing system 200 may be used to implement thesystem 100 in some examples. The computing device 218 may be used toimplement the computing device 114 in some examples. The processor(s)206 may be used to implement the processor 102 in some examples. In someexamples, the cloud computing device 230 may implement the cloudcomputing device 110. In some examples, light source 208 may implementlight source 106 shown in FIG. 1.

In some examples, computing device 218 may be included in a mobiledevice 222, such as a smart phone, cell phone, gaming device, or tablet.In some examples, the computing device 218 may be implemented wholly orpartially using a computer, a server, television, or a laptop. In someexamples, the spectrometer 210 may be an accessory device that connectswith the computing device 218 and/or mobile device 222. In otherexamples, the spectrometer 210 may be an integral element of thecomputing device 218 and/or mobile device 222, for example, a sensor ona smart phone. In some examples, such as the one shown in FIG. 2, thelight source 208, camera 212, and/or display 204 may be integralelements of the mobile device 222 in communication with the computingdevice 218, for example, a camera, a flash, and a touch screen of asmart phone.

In some other examples, the processor 206 may be implemented using oneor more central processing units (CPUs), graphical processing units(GPUs), application specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), and/or other processor circuitry. Insome examples, the processor(s) 206 may be in communication with amemory 216 via a memory controller 220. In some examples, the memory 216may be volatile memory, such as dynamic random access memory (DRAM). Thememory 216 may provide information to and/or receive information fromthe processor(s) 206 and/or computer readable medium 224 via the memorycontroller 220 in some examples. While a single memory 216 and a singlememory controller 220 are shown, any number may be used. In someexamples, the memory controller 220 may be integrated with theprocessor(s) 206.

The computing device 218 may include a computer readable medium 224. Thecomputer readable medium 224 may be implemented using any suitablemedium, including non-transitory computer readable media. Examplesinclude memory, random access memory (RAM), read only memory (ROM),volatile or non-volatile memory, hard drive, solid state drives, orother storage. The computer readable medium 224 may be accessible to theprocessor(s) 206 and/or memory controller 220. The computer readablemedium 224 may be encoded with executable instructions 228. Theexecutable instructions 228 may be executed by the processor(s) 206. Forexample, the executable instructions 228 may cause the processor(s) 206to analyze an acquired image to extract a measured spectrum from theimage. In some examples, the executable instructions 228 may cause theprocessor(s) 206 to correct the measured spectrum extracted from animage or acquired from spectrometer 210, for example, to correct foreffects of ambient light. In some examples, the executable instructions228 may cause the processor(s) 206 to provide commands or other controlsignals to the light source 208, the camera 212, the display 204,spectrometer 210, and/or other components of the computing device 218,such as the memory controller 220.

The computer readable medium 224 may store data 226. In some examples,the data 226 may include an image received from camera 212 and/or ameasured spectrum extracted from the image, a spectrum acquired byspectrometer 210, a corrected spectrum generated by the processor(s)206, a database of reference spectra, a product recommendation, adiagnosis, and/or user feedback. While a single medium is shown in FIG.2, multiple media may be used to implement computer readable medium 224.

The computing device 218 may be in communication with the display 204that is a separate component (e.g., using a wired and/or wirelessconnection) or the display 204 may be integrated with the mobile device222. In some examples, the display 204 may display data such as outputs(e.g., results) generated by the processor(s) 206, and/or imagesacquired by the camera 212. Any number or variety of displays may bepresent, including one or more LED, LCD, plasma, or other displaydevices.

In some examples, the user interface(s) 214 may receive inputs from auser 202. User inputs may include, but are not limited to, desiredresolution, selection of one or more region of interests, tastes andpreferences, and desired output (e.g., diagnosis, productrecommendation). Examples of user interface components include akeyboard, a mouse, a touch pad, a touch screen, and a microphone. Insome examples, the display 204 may be included in the user interface(s)214. In some examples, the processor(s) 206 may implement a graphicaluser interface (GUI) via the user interface(s) 214, including thedisplay 204. For example, the user 202 may review an image of asubject's face or other surface on the display 204, which may be a touchscreen in some examples, and circle an area of interest on the image. Insome examples, such as the one shown in FIG. 2, the subject may be theuser 202. In some examples, the processor(s) 206 may communicateinformation, which may include user inputs, data, images, and/orcommands, between one or more components of the computing device 218 andexternal apparatuses such as a spectrometer 210 that may generate ameasured spectrum of the surface. In other examples, the userinterface(s) 214 may be coupled with a recommendation engine accordingto a recommendation model to provide a recommendation of product. Theproduct recommendation may be based on the user's tastes andpreferences, and a determined match of the corrected spectrum generatedby the processor(s) 206.

In some examples, the light source 208 may emit light for acquiring animage with the camera 212 and/or spectrometer 210. In some examples, thelight source 208 may include a visible light source, an infrared lightsource, an infrared flood illuminator, an ultraviolet light source,and/or a broad spectrum light source (e.g., includes two or more ofvisible light, infrared light, and ultraviolet light). In someembodiments, the light source may be polarized. In some examples, thelight source 208 may include an LED, a xenon flash tube, and/or otherlight source. In some examples, the light source 208 may be includedwith the camera 212. In some examples, the light source 208 may emitlight responsive to a command provided by the camera. 212 and/or one ormore commands provided by the processor(s) 206.

In some examples, camera 212 may be used to acquire an image of asurface, such as surface 112 shown in FIG. 1. In some examples, camera212 may include a charged-couple device (CCD), complementary metal oxidesemiconductor (CMOS) sensor, and/or other light sensor. As described inmore detail with reference to FIG. 6, the processor(s) 206 may extract aspectrum from the image acquired by the camera 212. In some examples, inaddition or instead of acquiring an image with camera 212, spectrometer210 may acquire a spectrum of the surface. In some examples,spectrometer 210 may include a CCD, a light dependent resistor, aphotodiode, a phototransistor, and/or other light sensor.

In some examples, the user 202 may capture an image of a surface usingthe camera 212. The processor(s) 206 may extract a measured spectrumfrom the image and generate a corrected spectrum to remove ambient lightinfluence on the measured spectrum. In some examples, ambient lightinfluences may be removed by white balancing, which is described in moredetail with reference to FIG. 6. In some examples, the user interface(s)214 may prompt the user 202 to capture a second image of a white card orother “control” under the same lighting conditions the surface image wasacquired. The processor(s) 206 may extract a second measured spectrumfrom the second image and analyze the second measured spectrum tocompensate for ambient light influence from the second measured spectrumand generate a second corrected spectrum. In some examples, whitebalancing may be applied on the second image to account for the “colortemperature” of the light source 208 and/or ambient light.

In another example, camera 212 may acquire multiple images of theilluminated surface at once. The processor(s) 206 may compare eachmeasured spectrum of the multiple acquired images and generate acorrected spectrum for each measured spectrum. The processor(s) 206 maygenerate an average spectrum of the corrected spectra of the multipleimages.

The corrected spectrum (or averaged corrected spectrum in the exampleswhere multiple spectra are acquired) generated from the measuredspectrum (or spectra) by the processor(s) 206 by white balancing and/orother method may be compared with one or more reference spectra in adatabase of reference spectra, which may be stored as data 226 in thecomputer readable medium 224 in some examples. In some examples, one ormore of the spectra may each correspond to a different product (e.g.,different paint color, different foundation color). In other examples,one or more of the spectra may each correspond to a condition (e.g.,rosacea, melanoma). In some examples, the processor(s) 206 may determinethe reference spectrum that has the closest match to the correctedspectrum. Techniques for comparing and/or determining closest match aredescribed in more detail with reference to FIG. 4. Based on thecomparing, the processor(s) 206 may output the product and/or condition(e.g., diagnosis of the condition) corresponding to the closest matchingreference spectrum as a result. In some examples, a productrecommendation and/or diagnosis may be based, at least in part, on theresult, user preferences, user feedback, or a combination thereof.

In some examples, the processor(s) 206 may implement a machine learningmodel that is trained to provide the product recommendation and/ordiagnosis based on the corrected spectrum and/or closest matchingreference spectra(um). In some examples, the machine learning model mayprovide the product recommendation and/or diagnosis based, at least inpart, on user feedback received with respect to a prior productrecommendation and/or diagnosis. In some examples, the machine learningmodel may be implemented by executable instructions 228 executed by theprocessor(s) 206. In some examples, the machine learning model, or aportion thereof, may be implemented in hardware included withprocessor(s) 206.

In some examples, the processor(s) 206 may be communicatively coupled toa cloud computing device 230 through the user interface(s) 214. Theprocessor(s) 206 may provide the images and/or spectra (e.g., measured,corrected) to the cloud computing device 230. The cloud computing device230 may generate a corrected spectrum, compare the corrected spectrum toa database of reference spectra stored on the cloud computing device230, provide a result based on the comparison (e.g., productrecommendation, diagnosis), and/or make an inference to generate aresult based at least in part on a machine learning model implemented bythe cloud computing device 230. The cloud computing device 230 mayprovide the corrected spectrum, one or more reference spectra,diagnosis, and/or recommended product(s) to the processor(s) 206. Thecorrected spectra, product recommendation, and/or diagnosis may beprovided on display 204 in some examples. In some examples, the cloudcomputing device 230 may include a database of products and/orconditions in a computer readable medium/media (not shown). Thisarrangement may be desirable in some applications, for example, when thecomputing device 218 may have limited computational ability, such as ifcomputing device 218 is included with a compact mobile device 222. Thisarrangement may be more convenient for when the machine learning modelis dynamically trained and/or the spectral database is dynamicallyupdated.

FIG. 3 is an example of a mobile device 300 in accordance with examplesof the present disclosure. In some examples, mobile device 300 may beused to implement mobile device 222. The mobile device 300 may include adisplay 302. The display 302 may provide the GUI implemented by themobile device 222. For example, the display 302 may provide a diagnosis306, a measured spectrum 308 and/or a corrected spectrum 310. In someexamples, the mobile device 300 provides and displays on display 302 adiagnosis representative of a. comparison between the corrected spectrum310 and a database of spectra. The display 302 may include a productrecommendation 304 based at least in part on the diagnosis 306.

Optionally, in some examples, the display 302 may provide an image 312of a surface (e.g., surface 112) acquired by a camera (e.g., camera 212)of the mobile device 300. In some examples, a user (e.g., user 202), maybe able to select a region of interest (ROI) 314 within the image 312.For example, if display 302 is a touch screen, the user may tap orcircle a portion of image 312 to select the ROI 314. In some examples,the measured spectrum may be extracted only from data in the image 312in the ROI 314.

FIG. 4 illustrates example spectral plots 400 in accordance withexamples of the present disclosure. The spectral plots 400 may begenerated by and/or stored in a computing device, such as computingdevice 110 and/or 218, which may be included on a mobile device such asmobile device 222 and/or mobile device 300. In some embodiments, thespectral plots 400 may be displayed to a user, for example on display204 and/or 302. In FIG. 4, spectral plot (a) illustrates a correctedreflectance spectrum 404 over a range of wavelengths. Examples of therange of the wavelengths include an ultraviolet spectrum (UV), a visiblespectrum, and/or an infrared spectrum. The UV spectrum includes awavelength range generally from about 10 nm to about 400 nm. The visiblespectrum includes a wavelength range generally between about 380 nm andabout 760 nm. The infrared spectrum generally includes a wavelengthrange of about 700 nm to about 1 mm.

As described herein, for example with reference to FIGS. 1 and 2, acorrected spectrum (e.g., a corrected spectrum calculated by processor102 and/or processor 206) may be compared to one or more referencespectra from a database of reference spectra by a computing device, suchas computing device 110 and/or 218. In some examples, each spectrum maycorrespond to a product (e.g., paint color, foundation shade). Spectralplots (b) and (c) are examples of a comparison between the correctedreflectance spectrum 404 and reference spectra 402, 406 from a databaseof reference spectra. Methods of comparison may include a regressionanalysis such as least mean squares, least squares, or total leastsquares. In some examples, the reference spectrum having the closest fitto the corrected spectrum of the surface may be used to identifycandidate product(s) and provide a product recommendation. In thisexample, spectral plot (b) illustrates the corrected spectrum 404 incomparison to a first reference spectrum 402 and spectral plot (c) isthe corrected spectrum 404 in comparison to a second reference spectrum406. In the example shown in FIG. 4, the first spectrum 402 has a closerfit to the corrected spectrum 404 than the second spectrum 406. Thus, aproduct corresponding to the spectrum of 402 may be selected as therecommendation. In an example where the user seeks a productrecommendation for their skin (e.g., a foundation), the recommendedproduct(s) may have a spectrum that is the closest match to the spectrumof the user's skin tone.

Additionally or alternatively, instead of finding the reference spectrawith the best fit over the entire range of wavelengths, the comparisonmay be over a portion of the wavelengths. Finding the closest match overonly a portion of the spectrum may be suitable when only a portion ofthe spectrum will influence the appearance of the product. For example,spectral properties of a product in the UV and/or far IR regions mayhave little to no influence on how the product will appear to a user.Accordingly, in some examples, these regions of the spectrum may beignored when calculating the reference spectrum that is the best fit tothe corrected spectrum 404.

In other examples, the user may seek recommendations for decor based ona color of a wall. Based on the corrected spectrum of the wall, thecandidate products may include fabrics and/or other products of colorsthat match the wall. In these examples, the corrected spectrum may becompared to reference spectra as discussed above. Additionally oralternatively, rather than providing a matching (e.g., same) color,colors that are aesthetically pleasing or complement the color of thewall may be provided as product suggestions. For example, if the wall isdetermined to be a bright tone that captures attention due to its stronghue and/or brightness (e.g., red, yellow, orange), the recommendationmay be a color that balances the strong hue and/or brightness (e.g.,tan, gray). In these examples, the closest matching reference spectrummay be associated with a database of complementary colors and/orproducts in complementary colors. Alternatively, the corrected spectrummay be analyzed to determined relative or absolute intensities indifferent spectral ranges (e.g., red, blue, green). These intensitiesmay be compared to reference tables of intensities that are correlatedto databases of complementary colors. In another example where the useris looking for a recommendation of clothing articles, the user mayacquire an image of an article of clothing and the user may receiverecommendations for other apparel or accessories that may go well withthe clothing article in the image based on the color. Examplerecommendations may include a pair of shoes, earrings, ties, sunglasses,etc.

FIG. 5 is a flow chart of a method 500 in accordance with examples ofthe present disclosure. The method in FIG. 5 illustrates a method whenuser feedback is included in the database of product recommendationsand/or diagnoses updated based on the user feedback. In some examples,all or a portion of method 500 may be performed by a computing device,for example, computing device 114 in FIG. 1 and/or computing device 218in FIG. 2. In some examples, all or a portion of the method 500 may beperformed by a machine learning model implemented by processor 102 inFIG. 1 processor(s) 206 in FIG. 2, cloud computing device 110, and/orcloud computing device 230.

At block 502, “receiving user feedback” may be performed. In someexamples, the display (e.g., display 204 and/or display 302) may promptthe user to provide user feedback on the product recommendation(s). Inan example, the user feedback may be received by a user interface, suchas user interface(s) 214 illustrated in FIG. 2. In some examples, theuser feedback may include text, images, and/or sound data. In someexamples, the user feedback may be received from a mobile device, suchas mobile device 222, and/or a computing device, such as computingdevice 114 or computing device 218. The user interface(s) 214 maytransmit the user's input to a processor (e.g., processor 102 and/orprocessor(s) 206) and/or a cloud computing device (e.g., cloud computingdevice 110 and/or cloud computing device 230).

At block 504, “ranking the user feedback” may be performed. In someexamples, a rank may indicate a level of satisfaction of the productrecommendation and/or diagnosis provided by the user. At block 506,“classifying the user feedback,” may be performed. In some examples, aclassification may indicate one or more factors included in the userfeedback. For example, a user may be satisfied with the accuracy of thediagnosis, but unsatisfied with the price range of the productrecommendation. In some examples, the classifying may be a rules-basedclassification. In some examples, block 506 may be performed beforeblock 504. In some examples, blocks 504 and 506 may be performedsimultaneously.

At block 508, “providing the user feedback to a machine learning model”may be performed. The user feedback provided to the machine learningmodel may include the ranked user feedback in some examples. That is,the user feedback provided to the machine learning model may be modifiedfrom the data originally received at block 502. In some examples, theuser feedback may be used as a training data set to train the machinelearning model. Optionally, at block 510 “training the machine learningmodel” may be performed. The machine learning model may be trained withthe training data set.

As such, the user feedback may be reflected in future productrecommendations and/or diagnoses. In some examples, a productrecommendation of a lower price range may be provided as a substituteproduct for the user who finds the result accurate, but the recommendedproduct too pricey. In some other examples, the user interface mayperiodically ask the user to complete a survey. The survey may includequestion including ratings on the product recommendation and/or accuracyof diagnosis, user's preferences on additional features of the product,e.g., Sun Protection Factor (SPF) products, highly concealingfoundation, hypoallergenic paint, etc. Accordingly, the user may trainthe machine learning model to recommend products with the feedback theuser provides to the user interface.

In some examples, the machine learning model may include a neuralnetwork. In some examples, neural network may be a convolutional networkwith two dimensional or three dimensional layers. The neural network mayinclude input nodes that receive input data (e.g., the correctedspectrum, reference spectrum, combinations thereof, and/or portionsthereof). In some examples, the input nodes may be organized in a layerof the neural network. The input nodes may be coupled to one or morelayers of hidden units by one or more weights. In some examples, thehidden units may perform operations on one or more inputs from the inputnodes based, at least in part, with the associated weights between thenodes. In some examples, the hidden units may be coupled to one or morelayers of hidden units by weights. The hidden units may performoperations on one or more outputs from the hidden units based, at leastin part, on the weights. The outputs of the hidden units may be providedto an output node to provide a result (e.g., product recommendation,diagnosis). Of course, a neural network is provided merely as an exampleand one or more other machine learning models may be used (e.g.,decision trees, support vector machines). In some examples, the machinelearning model may include different machine learning models fordifferent applications. For example, there may be a machine learningmodel for cosmetic applications (e.g., determining cosmetic productrecommendations) and a separate machine learning model for otherapplications (e.g., determining coordinating apparel/accessories),

In some examples, the machine learning model may be trained by providingone or more training data sets. The machine learning model may betrained by the computing device (e.g., computing device 114, 222, cloudcomputing device 110, 230) used to make inferences with the machinelearning model in some examples. In some examples, the machine learningmodel may be trained by another computing device to determineappropriate machine learning model to implement, weights, nodearrangements or other model configuration information, and the trainedmachine learning model may be provided to the computing device used tomake the inferences.

In some examples, the machine learning model may be trained usingsupervised learning techniques. In some examples, training data mayinclude a set of inputs x, each associated (e.g., tagged) with a desiredresult y. Each input x may include one or more values for one or morespectra. For example, one input x may include a spectrum associated witha result y that is a reference spectra associate with a paint color.Based on the training data set, the machine learning model may adjustone or more weights, number of layers, or other components of the model.The trained machine learning model may then be used to make inferenceson inputs x (that are not associated with desired results) to generateresults y. In some examples, the machine learning model may be trainedusing semi-supervised and/or unsupervised techniques. In these examples,data sets may not include a desired result associated with every input.

In some examples, such as the one described in FIG. 5, the machinelearning model may be dynamically trained. That is, even if the machinelearning model is initially trained by another device, the machinelearning model may continue to adjust based on new data. For example,new products and/or user feedback may cause the machine learning modelto adjust.

FIG. 6 is a flowchart of a method in accordance with examples of thepresent disclosure. At block 602, the mobile device irradiates a surfacewith a light source, e.g., light source 106 in FIG. 1 and/or lightsource 208 in FIG. 2. At block 604, the mobile device acquires an imageof the irradiated surface with a camera as shown in camera 104 in FIG. 1and camera 212 in FIG. 2.

At block 606, a processor, such as processor 106 and/or processor 206extracts a measured spectrum representative of a color of the surfacefrom the image. In some examples, the processor may extract a measuredspectrum from the image by decoding the color of the surface withred-green-blue (RGB) weighting and assigning an RGB color code for thecolor of the surface. Histograms of red, green, and blue pixels may begenerated for an image. The color of the surface may be determinedbased, at least in part, on the concentration of pixels. For example, ifthere is approximately the same number of red, green, and blue pixels,the processor may assign an RGB color code indicating no one color ispredominate.

Additionally or alternatively, the processor may extract a measuredspectrum by applying MPEG 7 global descriptors. MPEG 7 globaldescriptors define various color descriptors representing differentaspects of color features including the representative colors, the basiccolor distribution, the global spatial distribution of colors, the localspatial distribution of colors, and the perceptual temperature feelingof an image. There are five color descriptors to represent differentaspects of color features: DominantColor for representative colors,ScalableColor for basic color distribution, ColorLayout for globalspatial distribution of colors, ColorStructure for local spatialdistribution of colors, and ColorTemperature describing the perceptualtemperature feeling of an image. In addition, the GoFGoPColor descriptoris defined as an extension of ScalableColor to groups of frames orpictures. Three supporting tools are defined: ColorSpace, ColorQuantization, and IlluminationInvariantColor. All the descriptors andtools are applicable to arbitrarily shaped regions. In correcting themeasured spectrum, a signal to noise ratio of the measured spectrum maybe optimized by using a filter and/or averaging the measured spectra.

Thus, the RGB color code, histogram, and/or MPEG 7 global descriptorsmay be used to generate a measured spectrum in some examples. In someexamples, this may be performed by machine learning. The machinelearning model may be trained similar to as described above withreference to FIG. 5. For example, the machine learning model may beprovided with RGB color codes, histograms, and/or MPEG7 globaldescriptors associated with spectra acquired by a spectrometer. However,in other examples, other methods of generating a spectrum from the RGBcolor code, histogram, and/or MPEG 7 global descriptors may be used. Inother examples, the measured and corrected spectrum are maintained inthe RGB color code, histogram, and/or MPEG 7 global descriptor formatand the spectra of the database of reference spectra is also in RGBcolor code, histogram, and/or MPEG 7 global descriptor format.

At block 608, the measured spectrum is adjusted by the processor byremoving ambient light influence on the measured spectrum to a correctedspectrum is generated. In some examples, ambient light influences may beremoved by white balancing. In some examples, the user may be promptedto take an image of a white card as a white reference under the samelighting environment as that of the surface. The processor may extract ameasured spectrum of the white card representative of ambient lightinfluence (Ia) and a measured spectrum of the image of the surface. Togenerate a corrected spectrum of the surface, the processor may subtractthe measured spectrum of the white card from the measured spectrum ofthe image of the surface.

At block 610, the corrected spectrum may be compared with a database ofreference spectra to determine a reference spectrum having a closest fitto the corrected spectrum by the processor. For example, as discussedwith reference to FIG. 4. At block 612, a result may be provided based,at least in part, on the spectrum having the closest fit. In someexamples, the result may be provided via a GUI on a display, forexample, display 204 and/or display 302.

Optionally, in some examples, a flood illuminator or other light sourcemay illuminate the surface of which an additional image is acquired todetect additional surface properties including texture andsub-properties. Texture analysis may be performed to adjust for thetexture from the color of the surface. Examples of texture may includeshininess of the skin. White balancing may be applied on the additionalimage to account the “color temperature” of the light source. A colorspectrum may be analyzed based on the image of the surface and theadditional image taken with the flood illuminator. A diagnosis orrecommendation of product may be provided based on the color spectrum.

Optionally, in some examples, the corrected spectrum may be analyzed todetermine a shininess of the surface. In an example of the surface beingskin cells, shininess may indicate oiliness. A matte foundation may berecommended for oily skin. In some examples, the shininess of thesurface may be determined by a specular reflectance based on a ratio ofintensities of a light source used to illuminate the surface and lightreflected from the surface. For example, shininess may be determined bya ratio of an intensity of a reflected beam and an intensity of anunpolarized light source. The surface may be determined to be shiny ifthe intensity of the reflected light is greater than a threshold value.In some examples, the threshold value may be some percentage of theintensity of the light source. A surface may be determined to be matteif the intensity of the reflected light is less than the thresholdvalue.

FIG. 7 is a flowchart of a method in accordance with examples of thepresent disclosure. The method in FIG. 7 illustrates a method for when aspectrometer, such as a spectrometer 104 in FIG. 1 and/or spectrometer210 in FIG. 2, is used to acquire a spectrum of a surface.

At block 702, a mobile device illuminates a surface having a desiredcolor using a light source, e.g. light source 106 in FIG. 1 and/or lightsource 208 in FIG. 2 The light source may be provided by the mobiledevice, such as mobile device 222 in FIG. 2 or mobile device 300 in FIG.3.

At block 704, the processor, as illustrated by processor 102 and/orprocessor(s) 206, of the mobile device may obtain a spectrum of signalsreflected from the surface having the desired color as explained inreference to FIGS. 1 and 2. In some examples, the spectrum of signalsmay be obtained by a spectrometer coupled to the mobile device asexplained in reference to FIG. 2.

At block 706, the mobile device may access stored representations ofcandidate product samples. The stored representations may be a localmemory, implemented by memory 216, or a remote memory including a cloudmemory, etc. Examples of the cloud memory may be illustrated in cloudcomputing device 110 in FIG. 1 and/or cloud computing device 230 in FIG.2.

At block 708, the processor compares the spectrum of signals obtained bythe spectrometer and selects at least one of the candidate productsamples as a match to the desired color based on a comparison betweenthe spectrum and the stored representations. In the example where thecandidate product samples include makeup foundation, the surface of thedesired color may be skin. In another example where the candidateproduct samples include paint samples, the surface having the desiredcolor may be an object (e.g., a wall, car) or a portion of an image.Optionally, in some examples, the user may provide user feedback withrespect to the candidate product samples as described in FIG. 5. In someexamples, the user feedback may influence future selection of candidateproduct samples.

As disclosed herein, in a variety of applications (e.g., cosmetics,interior design, fashion), a mobile device may analyze surface color andprovide a diagnosis of the surface and/or product recommendation basedon analysis of the surface. Accordingly, the devices, systems, methods,and apparatuses of the present disclosure may allow users to acquiredata on surface colors and receive a recommendation based, at least inpart, on the surface color without expensive and/or specializedequipment.

Of course, it is to be appreciated that any one of the examples,embodiments or processes described herein may be combined with one ormore other examples, embodiments and/or processes or be separated and/orperformed amongst separate devices or device portions in accordance withthe present systems, devices and methods.

Finally, the above-discussion is intended to be merely illustrative andshould not be construed as limiting the appended claims to anyparticular embodiment or group of embodiments. Thus, while variousembodiments of the disclosure have been described in particular detail,it should also be appreciated that numerous modifications andalternative embodiments may be devised by those having ordinary skill inthe art without departing from the broader and intended spirit and scopeof the present disclosure as set forth in the claims that follow.Accordingly, the specification and drawings are to be regarded in anillustrative manner and are not intended to limit the scope of theappended claims.

What is claimed is:
 1. A device comprising: a light source configured to illuminate a surface; a camera configured to acquire an image of the illuminated surface; and a processor configured to: extract a measured spectrum from the image; generate a corrected spectrum by at least partially compensating for ambient light influence on the measured spectrum; compare the corrected spectrum with reference spectra of a database of reference spectra to generate a result; and provide at least one of a product recommendation or a diagnosis based, at least part, on the result.
 2. The device of claim 1, wherein the comparison of the corrected spectrum with reference spectra of the reference spectra of the database includes using a regression analysis method comprising at least one of least mean squares, least squares, or total least squares.
 3. The device of claim 1, wherein the result includes a spectrum of the reference spectra having a closest fit to the corrected spectrum, a group of spectra of the reference spectra having the closest fit at different spectral ranges, or a combination thereof.
 4. The device of claim 1, further comprising a user interface configured to receive a preference from a user, wherein the user interface is coupled to a recommendation engine configured to provide the product recommendation based on the user's preference and the result.
 5. The device of claim 4, wherein the user interface is further configured to prompt the user to move the device to a new region of the surface and wherein the camera is further configured to take a second image of the new region, wherein the processor is further configured to extract a second measured spectrum from the second image of the new region, analyze the second measured spectrum to compensate for ambient light influence from the second measured spectrum and generate a second corrected spectrum, compare the second corrected spectrum with the database to generate a second result, and provide at least one of a second product recommendation or a second diagnosis based, at least in part, on the comparison between the corrected spectrum and the database and a second comparison of the second corrected spectrum and the database.
 6. The device of claim 1, wherein the camera is further configured to acquire a second image of a white reference under a same lighting environment as the illuminated surface, and wherein the processor is further configured to: extract a second measured spectrum from the second image; and subtract the second measured spectrum of the second image from the measured spectrum of the image to generate the corrected spectrum.
 7. The device of claim 1, wherein the camera is further configured to acquire multiple images of the illuminated surface, and wherein the processor is further configured to compute an average spectrum of the measured spectra of the multiple images and generate a second corrected spectrum based on the average spectrum, compare the second corrected spectrum with the database to generate a second result, and provide at least a second product recommendation or a second diagnosis based, at least in part, on the comparison between the second corrected spectrum and the database.
 8. The device of claim 1, wherein the database is stored in a cloud computing device in communication with the processor or a local memory included with the device in communication with the processor.
 9. The device of claim 1, wherein the processor is configured to implement a machine learning model to provide at least one of the product recommendation or the diagnosis, wherein the machine learning model is based at least in part, on user feedback.
 10. The device of claim 1, wherein the surface comprises skin cells, and wherein the product recommendation comprises a foundation that matches a skin tone of the skin cells.
 11. The device of claim 1, wherein the surface comprises a wall, and wherein the product recommendation comprises at least a decor that is complementary to the wall.
 12. The device of claim 1, further comprising a display, wherein the display provides at least one of the image of the surface, the product recommendation, or the diagnosis.
 13. The device of claim 1, wherein the light source is a visible light source or an infrared (IR) flood illuminator.
 14. The device of claim 1, wherein the device is a mobile device.
 15. A method comprising: irradiating a surface with a light source of a mobile device; acquiring an image of the irradiated surface; extracting a measured spectrum representative of a color of the surface from the image; correcting the measured spectrum, wherein correcting the measured spectrum comprises adjusting for ambient light influence on the measured spectrum to generate a corrected spectrum; comparing the corrected spectrum with a database of reference spectra to determine a spectrum of the reference spectra having a closest fit to the corrected spectrum; and providing a result based, at least in part, on the spectrum having the closest fit.
 16. The method of claim 15, wherein extracting comprises decoding the color with red-green-blue (RGB) weighting; and providing an RGB color code of the color of the surface.
 17. The method of claim 15, wherein extracting comprises analyzing the color spectrum further comprises applying MPEG7 global descriptors.
 18. The method of claim 15, wherein correcting the measured spectrum further comprises optimizing a signal to noise ratio of the measured spectrum.
 19. The method of claim 15, wherein removing the ambient light comprises subtracting a spectrum of a white reference from the measured spectrum or white balancing.
 20. The method of claim 15, further comprising analyzing the corrected spectrum to determine a shininess of the surface, wherein the shininess of the surface is determined by a ratio of an intensity of a reflected beam and an intensity of an unpolarized light, and wherein the result is based, at least in part, on the shininess of the surface.
 21. The method of claim 15, further comprising analyzing at least one of the corrected spectrum or the spectrum having the closest fit with a machine learning model to make an inference; and providing the result based, at least in part, on the inference.
 22. The method of claim 15, wherein comparing the corrected spectrum to the reference spectra includes comparing the corrected spectrum and the reference spectra across a range comprising an ultraviolet (UV) spectrum, a visible spectrum, and an infrared spectrum.
 23. The method of claim 15, wherein the result includes at least one of a product recommendation or a diagnosis.
 24. The method of claim 23, wherein the product recommendation is for a product having a spectrum complementary to the spectrum having the closest fit to the corrected spectrum, and wherein the product recommendation is based on a database of products based at least in part, on user feedback.
 25. The method of claim 15, further comprising: acquiring an additional image of the surface with a flood illuminator detect texture and sub-surface; performing texture analysis to adjust for the texture from color of the surface; applying white balancing on the additional image; analyzing a color spectrum based on the image and the additional image; and providing a recommendation of product or a diagnosis based, at least in part, on the analyzed color spectrum.
 26. A device comprising: a light source, wherein the light source is configured to illuminate a surface; a camera configured to acquire an image of the surface; an interface configured to couple a spectrometer, wherein the spectrometer is configured to generate a measured spectrum from the surface; a processor configured to: analyze the measured spectrum to compensate for an influence of ambient light from the measured spectrum and generate a corrected spectrum; and a cloud computing device in communication with the processor comprising a database of reference spectra, wherein the cloud computing device is configured to: compare the corrected spectrum with the database of reference spectra to generate a result; and provide at least one of a product recommendation or a diagnosis based, at least in part, on the result.
 27. The device of claim 26, further comprising a display configured to provide at least one of the image of the surface, the spectrum of the surface, and at least one of the product recommendation or the diagnosis.
 28. The device of claim 26, wherein the cloud computing device is further configured to implement a machine learning model to provide the product recommendation or the diagnosis, and wherein at least one of the product recommendation and the machine learning model is updated with user feedback.
 29. The device of claim 26, wherein the processor is further configured to determine a specular reflectance of the illuminated surface based at least in part on an intensity of a reflected beam and an intensity of an unpolarized light.
 30. The device of claim 29, wherein the surface is determined to be shiny if the intensity of the reflected beam is greater than the intensity of the unpolarized light. 